Caching, clustering and Aggregating usenet server

ABSTRACT

A cluster of data caching servers that provide UseNet service to customers. The cluster of data caching servers cache articles and data requested by customers after retrieving the articles and data from a backend server/server farm. The cluster of data caching servers are adapted to share their cache memories with the other clustered servers in the cluster such that, if an article is requested by a client and the article has already been stored in one of the clustered server&#39;s cache memories, there is no requirement to retrieve all the requested article from the backend server.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application for patent claims the benefit of priority from, andhereby incorporates by reference the entire disclosure of, co-pendingU.S. Provisional Application for patent Ser. No. 60/414,204, filed Sep.26, 2002.

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

The present invention relates to Usenet servers, and more specifically,the present invention relates to the storage and retrieval of articlesin a UseNet system.

2. Description of Related Art

UseNet server systems provide news services to clients. Conventionally,UseNet backend servers have been located remotely from the serviceproviders that provide news services requested by clients. Thus, due tothe remote location of the Usenet backend servers, news services havetraditionally been slow and require a large amount of bandwidth betweenthe UseNet service provider and the UseNet backend servers. This set upis both expensive and inefficient. Thus, what is needed is a UseNetserver capable of providing news services at an increased speed withreduced costs.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides a UseNet server systemthat operates as a pass-through caching Network News Transfer Protocol(NNTP) front-end system. No data or metadata need pass through theUseNet server system except as a result of a client request. In responseto client requests, the system will first check servers in a localcluster of servers to determine if the requested data is available. Ifthe requested data is available on a server in the local cluster ofservers, then the UseNet system will connect to the server and retrievethe requested data. Otherwise, the UseNet system will determine which ofa given number of unique backend server farms to connect to process therequest in order to retrieve the requested data. When the requested datais passing from the backend server to the client, the UseNet system maycache the data and meta data within the local cluster of servers at thesame time.

In certain embodiments, on a given host, any backend or cluster serverconnections established as a result of a request for data are maintainedand reused by other processes within the local server, as they areneeded. Depending on the configuration, an exemplary UseNet system canprovide a different subset of groups and retention dates based on thesource IP of the client and on other authentication information providedby the client.

In further embodiments, the authentication, authorization, andaccounting are handled by a Secure Sockets Layer (SSL) encryptedTransmission Control Protocol (TCP) session back to an authenticationserver. An authentication server is a separate server that handlesauthentication processes related to the UseNet services. Cached articlescan be stored in a compressed format in the local cluster servers. Thecached articles can also remain in a compressed format when transferredbetween two servers. An exemplary UseNet server system can transparentlyhandle most failures of the backend/cluster servers. In addition, theuse of raw (non-mirrored) disks is possible through failure detectionand mitigation of further use by the UseNet server system. Storage ofdata on the raw disks is done by using time-stamped buckets to provideneeded retention and overwriting of cached articles based on the age ofthe data. Recycling of a previously used bucket into a new time framedoes not wipe or erase the existing data until it is physicallyoverwritten with new data.

In still further embodiments, a UseNet server system can be run in sucha fashion that substantially none of the UseNet server software isstored on the server hardware. The software itself can be retrieved viaa secure SSL connection using a Challenge Handshake AuthenticationProtocol (CHAP) based authentication technique to ensure that onlyauthorized servers or systems may retrieve the software.

Embodiments of the exemplary clustered local servers allow theaggregation of multiple NNTP servers. The clustering of local serversenhances scalability because each clustered server is substantially thesame operationally. In addition, connection, caching, and passing, ofarticles between the clustered UseNet servers reduce theconnect/authenticate load on the backend servers. Throughover-the-network compression between server clusters (backend serverfarms and clustered servers), the system can conserve additional networkbandwidth above mere caching. Further, since the articles are alreadycached in the clustered local UseNet servers in a compressed format,there is no CPU time spent on decompressing from the cluster system andfurther no CPU time spent on recompressing on the local system as itcaches the article.

Embodiments may have an advantage of only reading from the masterservers on-demand, such that data is transferred from a backend serverfarm to a remote cluster of UseNet servers only if the data wasrequested by a client connected to the remote cluster of UseNet servers.The caching, clustering, and on-demand nature of exemplary software ofthe present invention enables a provision of service to a remote clusterof UseNet servers located at a client's location with assurances thatthe traffic the client will receive is less than or equal to a fullfeed.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed invention will be described with reference to theaccompanying drawings, which show important sample embodiments of theinvention and which are incorporated in the specification hereof byreference, wherein:

FIG. 1 is a block diagram of an exemplary UseNet server system.

FIG. 2 is diagram depicting an exemplary UseNet geographicconfiguration.

FIG. 3 is a flowchart illustrating exemplary steps for initializing thesystem of the present invention; and

FIG. 4 is a flowchart illustrating exemplary steps for enabling theclustering and caching process provided by the system of the presentinvention.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS OF THE INVENTION

The numerous innovative teachings of the present application will bedescribed with particular reference to the exemplary embodiments.However, it should be understood that these embodiments provide only afew examples of the many advantageous uses of the innovative teachingsherein. In general, statements made in the specification of the presentapplication do not necessarily delimit any of the various claimedinventions. Moreover, some statements may apply to some inventivefeatures, but not to others.

Referring to FIG. 1, an exemplary UseNet server system 10 is shown in ablock diagram. Customers or Clients 12A-12E are connected to the UseNetserver system 10 via standard NNTP connections. The customers 12 can bea few to any number of customers who are connected to the Internet, anInternet Service Provider (ISP), or by another means for connecting to aUseNet service.

A local cluster of servers (Gigacluster) 16 is established having aplurality of local servers (GC servers) 18A-18 n. Each GC server 18 canbe a commodity server or substantially a standard server having memory,processor(s), and hard drives. It is understood that one or more of theGC servers 18A-n may have additional components or upgraded devices whencompared with the other GC servers. For example, the different GCservers 18A-n each may different speed microprocessors, larger or fasterhard drives, upgraded electronics, etc when compared to each other.Although FIG. 1 only depicts three GC servers 18A-n there can besubstantially any number of GC servers 18A-n operating within theGigacluster 16. Each GC server 18 is connected or in electrical oroptical communication 19 with each one of the other GC servers 18A-n.

The Gigacluster 16 is connected to or is adapted to be in electrical oroptical communication with various backend servers 22A-n. In general aGigacluster 16 is connected via a TCP/IP connection(s) 24 to the backendservers 22A-n. Each backend server 22 may each be a plurality of serverssuch that backend server 22A represents two or more servers (i.e., aserver farm). The backend servers 22A-n may be located in locations thatare distant or remote from the other backend servers 22A-n. Also eachbackend server may be located distant or remote from the Gigacluster 16.For example, backend server 22A may be located in Austin, Tex., backendserver 22B may be located in Baltimore, Md., and backend server 22C maybe located substantially in London, England. Other backend servers orbackend server farms 22A-N may be located anywhere in the world.

There may also be additional Gigaclusters 16′ that are connected to oneor more of the backend servers 22A-n. The additional Gigaclusters 16′may be remotely located from both the backend servers 22A-n and theGigacluster 16. A backend server may also be a Gigacluster.

The backend servers 22A-n receive and store articles for many UseNetuser groups. The size of the data storage space required in each of thebackend servers 22A-n depends on the amount of UseNet article retentionthat is desired. At the present time, a single day's worth (24 hours) ofUseNet articles fills, on the average, approximately 750 to 1,000Gigabytes of storage space on the backend servers 22A-n. In order toincrease the retention to more than one day's worth of UseNet articlepostings, the size of the backend storage space on the backend servers22A-n must increase according to the number of days or the amount oftime retention is desired. Furthermore, embodiments of the presentinvention allow each UseNet client to specify the amount of retentiontime or the amount of retention space desired for their UseNet groups.For example, some UseNet user groups or Use Net clients may want to havea month's worth of retention for their specific UseNet user group(s),others may want more or less retention time or storage space.

As the activity of the UseNet increases so does the number of UseNetarticles and postings. To date activity on the UseNet has increasedsteadily since the UseNet's beginnings. It is expected that the storagerequirements on the backend servers 22A-n will increase from the present750 Gigabytes per day to more than 1000 Gigabytes per day for eachbackend server (farm) over the next year. The backend servers 22A-n willrequire continuous storage upgrades in order to provide enough articleretention time that satisfies customers, clients, and UseNet users. Theoutput bandwidth from the backend server 22A-n necessary on the TCP/IPconnections 24 between the Gigacluster 16 and backend servers 22A-nbecomes insufficient to handle the necessary data rate of more than 200to 300 Mbps. Thus, as the amount of data storage is increased on atleast one of the backend servers 22A-n, then the amount of data thatneeds to be transferred between the backend servers 22A-n and aGigacluster 16 also increases due to the growing needs of users andclients. As a result, the data connection between each of the backendservers 22A-n and the Gigacluster becomes a limiting factor. All of thenecessary data cannot be transferred over the TCP/IP connection 22resulting in data retrieval delays. The end result on the UseNet isdelays to the client and user. The present day backend servers cannotmove the amount of data requested by today's UseNet users in a timelyfashion. It is noted that each backend server (farm) 22A-n generallystores substantially the same information as the other backend serversor the backend servers are divided up into several server farms eachcontaining a subset of the same information.

Exemplary embodiments of the present UseNet server system use theservers (GC servers 18A-n) in the Gigacluster 16 to store or cachearticles that were recently requested by a customer 12 and that wereretrieved from a backend server 22. By retrieving articles from one ormore backend servers 22A-22 n, the Gigacluster 16 and its associated GCservers 18A-n transparently aggregate multiple backend servers 22A-n,wherein each backend server is responsible for a distinct subset ofUseNet groups.

Furthermore, by caching data retrieved from the back end servers andhaving the cached data available to customers upon request, the GCservers 18A-n reduce the load on the backend servers 22A-n and theTCP/IP buses' 24 bandwidth between the backend servers 22A-n and theGigacluster 16.

Each GC server 18 is in communication 19 with the other GC servers 18A-nwithin the Gigacluster 16. The GC server 18A-n may communicate with eachother using one or more various or a combination of differenttechnologies including, but not limited to hard-wired technologies,network technologies, wireless technologies, optical technologies,satellite technologies, and other types of networking technologies andprotocols. The GC servers 18A-n may utilize an ethernet 19 or other formof electronic communication within their cluster 16. The electroniccommunication 19 between the GC servers 18A-n also allows the GC server18A-n, when the appropriate backend server is contacted via the TCP/IPconnection 24, to retrieve the article or data from a backend server 22.

When a connection to a backend server 22A or to another GC server 18Bconnection is made by a GC server 18A, the GC server 18A retains andmaintains the connection so it can be reused as needed in the future foradditional movement of stored articles or data to a customer 12.Retaining and maintaining connections between the various servers(backend and GC server 18A-n) saves time and conserves the various CPU'sresources necessary for creating and breaking down any given datacommunication connection between servers.

New UseNet articles or new data tends to be read or downloaded bycustomers much more often than older UseNet articles or data. By storingor caching recently retrieved articles or data in the GC server 18A-nwhen they are being retrieved from the backend, then, in general, neweror more popular articles are being stored on the GC server 18A-n. Theresult of storing the retrieved articles in the GC server 18A-n is thatretrieval of the more popular or frequently read articles and data donot use the bandwidth of TCP/IP connections 24 or the backend servers22A-n after the frequently requested article or data has been retrievedfrom the backend once. The technique decreases the time required toretrieve popular or frequently requested articles. This technique ofcaching previously retrieved articles on the GC server 18A-n alsodecreases and/or completely removes the delay seen by the customer whenretrieving data from UseNets having inadequate TCP/IP bandwidth betweenthe UseNet Servers and their associated backend servers.

Due to the limitations of the backend servers 22A-n being able to pushdata to the customers via the TCP/IP connections 24, the GC server 18A-nwith the cached or stored articles increase the throughput. The concernof any UseNet system is how much data in a given amount of time can beprovided to the customers.

A backend server 22A-n and a GC server 18 are somewhat different typesof servers. A backend server 22A, for example, may be very heavy instorage space and be able to move about 200 Megabits per second over aTCP/IP connection 24 to the Gigacluster 16. The storage space of abackend server is unusually expensive because of the storage capacityand the high quality of the drives. The large storage capacity and highquality are required due to the amount of usage the drives endure. Abackend server 22A or farm of servers 22A may have anywhere from twelveterabytes to at least 40 terabytes of disk space. In a present daybackend server farm 22 requires about 750 to 1000 Gigabytes of diskspace (storage) to store a single day's work of UseNet data. It isunderstood that the storage space on the servers may be hard drive,disk, optical, holographic, tape, magnetic, bubble, solid state or anyother applicable storage media. As UseNet usage continues to growadditional hard drives must be added to the backend server or serverfarm just to maintain the same amount of article retention time. As aresult, backend servers tend to be one of the most expensive types ofservers. Backend servers are heavyweights when it comes to storagecapacity, but are lightweights when it comes to being able to move alltheir stored data around. One can say that the ratio of data capacity tothe amount of data that can be moved per second is a large number. Forthe smaller example backend server discussed above, the ratio is 4000Gigabytes to 200 Megabits/second. Thus, there is a 20,000:1 relationshipof storage of customer traffic. This ratio will increase with the sizeof the backend server 22 or server farm 22.

On the other hand, each GC server 18 can, in general, be a commodityserver. In the present day, a commodity server or standard general useserver is much less expensive than a storage-capacity-heavy backendserver. In the present day, a commodity server can move data at about200 Mbps (similar to a backend server). A present day commodity serverhas between about one half of a terabyte to two and a half terabytes ofdisk memory. Thus the ratio of data capacity to the amount of data thatcan be moved per second is a smaller and more balanced ratio of 2.5terabytes to 200 Mbps (12500:1) than the ratio of an exemplary backendserver. This is a much lower ratio than the backend server's ratio of20,000:1. Thus, a GC server 18 is more capable of quickly finding datain its memory and moving all or a large percentage of the data stored inmemory over time than a backend server 22.

An exemplary Gigacluster 16 is formed such that each GC server 18 hasabout 2.5 terabytes of storage and each GC server 18 offers about 200Mbps of data throughput to the many customers using the UseNet. If thereare, for example, 20 GC servers 18A-n in the Gigacluster 16, thencollectively there is 50 terabytes of storage for storing cachedarticles and data. Furthermore, the GC server 18A-n collectively havethe capacity of moving 200 Mbps times 20 servers equaling about 4000Mbps (4 Gbps) of data to the customers.

Since the GC server 18 is caching or storing articles or data not foundin any of the other GC server 18A-n within the Gigacluster 16, the newor most popular articles and data tend to be stored in the GC server18A-n after being retrieved from the backend. Thus, the GC server 18A-ncan provide, when requested, the popular or new articles to thecustomers 12 without querying the backend servers 22A-n. This exemplarytechnique of clustering the GC server 18A-n frees up bandwidth betweenthe Gigacluster 16 and the backend servers 22A-n.

To further increase the efficiency of an exemplary UseNet system, thearticles or data can optionally be transmitted in a compressed formatwhen transferred between servers. The articles stored in the GC servers'cache can be stored in the same compressed format as they weretransferred in. Sending and storing articles in a compressed formatsaves transmit and receive time, decreases the amount of storage spacerequired for each article, provides a savings in CPU processing time inboth the transmitting and receiving server, and uses less bandwidth ofthe connection between servers.

Another important aspect of an embodiment of the present invention ishow retention of stored articles and data is maintained by the GC server18A-n as well as the backend servers 22A-n. Exemplary embodiments mayprovide different views of the UseNet to different customers dependingon the UseNet service required by the client or customer. For example,an exemplary UseNet system is able to provide a view of or access toonly preselected UseNet group's articles or data. That is, if there arefive thousand separate UseNet groups, a customer or client may only haveaccess to 100 of them.

Furthermore, an exemplary UseNet system may provide different amounts ofretention access to different clients or customers. One customer mayhave access to the full article and data retention maintained by theexemplary UseNet system. Another customer may have access to a lesseramount of article and data retention.

An exemplary UseNet system offers the design and implementation ofmultiple redundant backend servers 22A-n and networks. Thus, the failureof any single backend server is contained and transparently handled by asmooth failover of connectivity within a backend selection codemechanism.

An exemplary UseNet system does not necessarily need to operate as aprimary news-server and further does not need to take a constant feed ofarticles from any source. Instead, the UseNet system may be located at aclient location. When deployed in a client's location, in a client'snetwork facility the GC server 18A-n may be deployed as a secondarycache which connect into the core systems through the in-house primarycache systems located in the primary network facility.

An exemplary UseNet system uses an article and data storage mechanism,in the form of software, on each GC server 18 and each backend serverthat is commonly referred to as Just a Bunch of Disks (JBOD). JBODimplies that each article cache within each GC server 18 does not have aredundancy copy of the cached articles. However, because of the abilityof any GC server 18 to access an article from another GC server 18 inthe Gigacluster 16 via the ethernet 19 or access the backend servers22A-n for an article via the TCP/IP connection 24, then if a storage(i.e., hard drive, solid state memory) failure occurred in one GC server18A, then the GC server 18A can continue to function by contacting itsGigacluster peers (the other GC server 18A-n in the Gigacluster 16) orthe backend servers 22A-n to obtain articles requested by a customer 12.

Each memory device or hard drive for storing articles and data arelogically divided into smaller sized data storage units called spools.The spools are further assigned (on a dynamic basis) to store articlesand data dated within a constant time interval; when the spools areassigned a time interval, then the spools are said to be assigned tothat time interval's “bucket”.

Once a spool is assigned to a specific time-interval's bucket, the spoolwill only allow articles that fall within the specific time-interval tobe stored in the spool's drive space.

If an article arrives for which there are no spools to store the article(the time interval's bucket is empty or all spools in the bucket arefull) then the server containing the spool will attempt to recycle aspool from the oldest bucket in the same server.

-   -   a. If there are no non-empty time-interval buckets older than        the article, then the article is not cached in the GC server.    -   b. If there are non-empty time-interval buckets older then the        article, then a spool from the oldest non-empty time-interval        bucket is moved to the time-interval bucket is moved to the        time-interval bucket for the currently received article. This        action is known as “recycling a spool”.

When a spool is recycled from one time-interval bucket to a newertime-interval bucket the articles stored in that spool for the oldtime-interval bucket are not lost until they are physically overwrittenby new incoming articles.

Hard drives, by their very nature, are commonly failed hardwarecomponents. Further, each GC server 18 employed provides no software orhardware mirroring that protects the articles cached in a GC server 18.To prevent a failed hard drive from affecting the performance of anexemplary UseNet system, each GC server 18 continually monitors forread/write failures through normal operations on each drive. When anallowed threshold of failures for a drive has been reached, the GCserver 18 or a master server will tag, in a metadata file, the fact thatthe drive has failed and all further access to that drive willimmediately cease. The drive is treated as if it never existed.

When a hard drive is replaced, a detection method is employed by whichthe specific device can be re-enabled.

Each GC server 18 employs the use of a number of metadata pieces ofinformation that would not allow proper system functionality if thatdata were to be lost by a sudden loss of a hard drive. For thismetadata, each GC server 18 employs the use of software mirroring toprotect from the failure of the GC server's hard drives.

Remotely deployed Gigaclusters 16′ and/or GC server 18A-n utilize a bootfloppy at the location of the system's bootable kernel. There is littleor no software stored on the actual deployed systems. The systemsoftware is specifically not located on any floppy or hard drive in anydeployed GC server 18. The software is obtained via an SSL connectionback to a distribution server where the remote system is authenticatedand authorized to retrieve the software. The software is then stored ina memory-only file system in an effort to prevent the theft of thesoftware itself.

The configuration file for the system is also retrieved on asystem-by-system basis via a remote SSL based distribution system. Theconfiguration file is generated on the fly and signed for authenticityusing a DSA signature algorithm.

FIG. 2 provides a diagram depicting an exemplary UseNet geographicalconfiguration. The exemplary use of cached data and articles withinspools on each GC server 18 with a Gigacluster 16 provides economicsavings to the UseNet system provider. FIG. 2 depicts a firstGigacluster 20 located in a first geographical location Texas1. A firstbackend server/server farm 22 is located in substantially the samegeographical location as first Gigacluster 20. A second Gigacluster 26is located in a second geographical location called Texas2. A thirdGigacluster 32 is located in a third geographical location called NewYork. As such, the first, second and third Gigacluster 20, 26 and 32 aredistant from each other. Also, the second and third Gigaclusters 26 and32 are geographically distant from the first backend server/server farm22.

A TCP/IP connection is adapted to connect the plurality of Gigaclusters20, 26, 32 to each other and to the first backend server 22 via, forexample, the internet or a worldwide computer system (not specificallyshown). Each of the three exemplary Gigaclusters is part of a UseNetprovider's UseNet system. Gigacluster 20 uses the first backend server22 for retrieving articles and data not located in the individualGigacluster 20. Gigacluster 26 uses the Gigacluster 20 for retrievingarticles and data not located in the Gigacluster 26. Gigacluster 32 usesthe Gigacluster 26 for retrieving articles and data not located in theGigacluster 32.

Assuming that each Gigacluster 20,26, 32 has been operating insubstantially a steady state for a period of time, then the GC server18A-n within each of the separate Gigaclusters will have cached the mostpopular and most recent articles and data.

At present, the TCP/IP 24 connection rate between each of theGigaclusters and the backend server 22 is about 120 Mbps (Megabits persecond). If the Gigaclusters did not cache the most requested articlesat each geographical location, then they would have to share the 120Mbps TCP/IP connection to receive each and every article requested bythe customers.

If each Gigacluster can push about 2 Gbps (Gigabits per second) to eachGigacluster's associated clients and customers, then the TCP/IPconnection could never keep up with the needs of each Gigacluster.Furthermore, a long haul TCP/IP connection becomes costly to the UseNetprovider when the TCP/IP bandwidth is used to its maximum extent.

In the exemplary embodiment, and via experimentation on the part of theinventors, it was discovered that the use of caching in the GC server18A-n of each Gigacluster of a UseNet system greatly decreased the need(and the cost) of using a TCP/IP connection to retrieve client requestedarticles from a backend server. In fact, in greater than 20% of therequests by customers, the GC servers 18A-n, of a given Gigacluster, didnot have to use the TCP/IP connection to retrieve an article or datafrom the backend servers 22A-n. Instead, one or more of the GC servers18A-n within the given Gigacluster had the requested article or data incache and was able to provide it to the requesting GC server 18, whichin turn provided the requested article or data to the customer.

In exemplary embodiments of the present invention and as a result ofexperimental operations of an embodiment of the present invention, thepercentage of requested articles and data found in cache on the GCserver 18A-n in a Gigacluster operating in a steady state ranged from atleast 20% to about 90%. In other words, the cache storage of theclustered GC server 18A-n within a given Gigacluster have a cache hitratio of about 20 to 90 percent when operating in steady state. A cachehit ratio can be defined as the amount of articles and data that can beprovided to the customers of a Gigacluster via the GC server 18A-ncompared to the amount of articles and data that can be retrieved fromthe Gigacluster. This excellent cache hit ratio greatly decreases thecosts of using a TCP/IP connection to a backend server farm, increasesthe efficiency of the UseNets system, and decreases the wait time UseNetcustomers experience during peak operating hours of a UseNet.

Exemplary embodiments of the present invention have provided bothunexpected results and have answered a long need for a solution tokeeping UseNet system costs under control without diminished service tocustomers and clients in the form of delays to their UseNet articlerequests. Furthermore, exemplary embodiments of the invention areenjoying immediate commercial success in that the UseNet service thathas recently begun employing one or more embodiments of the presentinvention has become more profitable and has found their UseNet servicein higher demand by customers because of the new level of serviceoffered at a competitive price.

Referring now to FIG. 3, to initialize an exemplary UseNet system, atstep 100, configuration files for each GC server 18 are read andvalidated for correctness and authenticity using DSA signaturealgorithm. At step 110, various subsystems are initialized. For example,if a GC server 18 is configured to use SSL for authentication, an SSLkey can be loaded and certified into memory, and SSL contexts can beinitialized. If any part of the initialization fails, an error can belogged and the process terminates. As another example, JBOD (cyclical)spool metadata can be initialized if cyclical spools are configured inthe system. To initialize JBOD spool metadata, each GC server 18 can runa test to determine if the cyclic spool shared memory metadata isalready configured. If not, the metadata from the cyclic spools can beread into a shared memory segment, verifying each spool with a checksum.If the checksum fails, the spool can be initialized as a new spool.Further updates to the cyclic spool metadata can be written to bothshared memory and disk. All reads of metadata are preferably from theshared memory segment, for speed purposes.

Thereafter, at step 120, the system can install signal handlers forvarious error conditions and termination requests. At step 130, thesystem can create listening sockets for NNRP services. Thereafter, atstep 140, the Gigacluster can wait for and accept a new connection fromthe various listening sockets. At step 150, the new connection forks anew process (The Child). The Parent process waits for the termination ofThe Child, and then closes the newly accepted socket and restarts thesocket accept loop at step 140. The Child Process forks (creating TheGrandchild) and immediately exits. The Grandchild closes all listeningsocket descriptors, and at step 160, initializes the customer or clientby initializing various settings for a generic, unauthenticated sessionand using the IP address from which the session was created toauthenticate the customer or client. Thereafter, at step 170, theGrandchild executes software for handling the connection.

To authenticate the customer or client, the system compares the IPaddress of the customer or client to a list of “always allowed chainers”addresses, and upon match, sets a session cookie and session tag to bothread “chainer”, sets permissions to allow READING and indicatesNO-IDLE-TIMEOUT during client command reading. If no username andpassword is provided, the system generates a connection authenticationrequest with just the IP address of the customer or client. Otherwise,the authentication request includes the IP address of the customer orclient, as well as the username and password provided by the customer orclient. If the GC server 18 is not already connected to the centralizedauthentication server, the GC server 18 connects to the authenticationserver.

Next, the GC server 18 negotiates SSL parameters if the configurationindicates SSL requirements. On any error or failure to connect, theprocess terminates. If no errors occur, the GC server 18 sends a requestto the authentication server and reads the response. If a GC serverfailure is detected in the transaction with the authentication serverand the connection has not already been retried, the system disconnectsfrom the authentication server and attempts to reconnect. If the secondattempt fails, the process terminates. If authentication is notsuccessful, the GC server 18 returns a message to the client indicatingsuch. Otherwise, the GC server 18 records the returned cookie and tagalong with the username and IP address of the session for future use inaccounting procedures.

Referring now to FIG. 4, at step 170, to handle the connection, at step200, the GC server 18 determines the allowed idle time when waiting fora command from the customer or client. At step 205, if the idle timeexpires before a command is read, the process terminates at step 210. Ifthe idle time does not expire before receipt of a command, at step 215,the GC server 18 reads a single command from the customer or clientsocket. At step 220, if the command is invalid, at step 225, the systemresponds with “500 syntax error or unknown command” and restarts theconnection handling process. At step 230, if the command requiresauthentication and, at step 235, the customer or client is consideredunauthenticated, at step 240, the system responds with “480authentication required” and restarts the connection handling process.

At step 245, if the command requires that a Use group be selected and,at step 250, no group has been selected by the remote client, at step255, the system responds with “412 no group selected” and restarts theconnection handling process. At step 260, if the command requirespermission and, at step 265, that permission is not granted to thecustomer or client, at step 270, the GC server 18 responds with anappropriate error and restarts the connection handling process.

Otherwise, at step 275, the GC server 18 records a snapshot of the totalbytes transferred to-date during this session (“bytes”). At step 280,using “GC—Client Dispatch Table” shown below, the system executes thespecified logic depending upon which command has been specified by thecustomer or client, as shown in more detail below. After execution ofthe specified logic, at step 285, the system captures a new snapshot ofthe bytes transferred and logs the command executed, the number of bytestransferred for the command, and the calculated rate of transfer. Thisprocess is repeated for the duration of the session connection.

At the termination of the session, if there is no authentication cookierecorded for this customer or client connection, the process terminates.However, if the cookie is a “chainer”, the GC server 18 zeroes out thecounters for the session and terminates the process. If there is anauthentication tag associated with the connection and there were someamount of bytes delivered to the customer or client, the GC server 18determines whether a connection to the centralized authentication serveris active. If so, the GC server 18 generates an accounting packetnecessary to associate the used bandwidth to the account information forthe client. If successful, the GC server 18 logs this information in a“real-time” logfile indicating that the account was handled inreal-time. Otherwise, the GC server 18 logs the information in a“post-processed” logfile for processing at a later point in time byexternal processes. Also, if the connection to the authentication serveris active, the GC server 18 sends a disconnect message with the cookie,IP, user, and tag of the connection.

The following is an exemplary client dispatch table. Exemplary processflows for each command are shown in more detail below.

GC - Client Dispatch Table Command Flow to Execute ARTICLE GC - ClientCommand ARTICLE AUTHINFO GC - Client Command AUTHINFO BODY GC - ClientCommand BODY DATE GC - Client Command DATE GROUP GC - Client CommandGROUP HEAD GC - Client Command HEAD HELP GC - Client Command HELP LASTGC - Client Command LAST LIST GC - Client Command LIST LISTGROUP GC -Client Command LISTGROUP MODE GC - Client Command MODE NEWGROUPS GC -Client Command NEWGROUPS NEWNEWS GC - Client Command NEWNEWS NEXT GC -Client Command NEXT POST GC - Client Command POST STAT GC - ClientCommand STAT XHDR GC - Client Command XHDR XOVER GC - Client CommandXOVER XPAT GC - Client Command XPAT QUIT GC - Client Command QUIT

GC—Client Initialization

-   -   1. Initialize various settings for a generic, unauthenticate        session.    -   2. Execute “GC—Client Authenticate” with the IP address from        which the session was created.

GC—Authentication System Initialization

-   -   1. If configured to use SSL for authentication, load SSL key and        certificate into memory and initialize SSL contexts. If any part        of the initialization fails, log an error and execute        “GC—Terminate Process”.

GC—Client Authenticate

-   -   1. If this is the first time “GC—Client Authenticate” has been        executed in this client process, register “GC—Auth Fini” for        running at process termination time for cleanup purposes.    -   2. Compare the IP address of the client to a list of “always        allowed chainers” addresses and upon match, set session cookie        and session tag to both read “chainer”, set permissions to allow        READING and indicate NO-IDLE-TIMEOUT during client command        reading. Exit this flow.    -   3. Build an authentication request under the following logic:        -   A. If no username and password is provided, generate a            connection authentication request with just the IP address            of the client.        -   B. Else, build an authentication request with the IP address            of the client, as well as the username and password provided            by the client.    -   4. If not already connected to the centralized authentication        server, connect to the server. Negotiate SSL parameters if the        configuration indicates SSL requirements. On any error or        failure to connect, execute “GC—Terminate Process”.    -   5. Send request to authentication server and read response.    -   6. If a system failure is detected in the transaction with the        authentication server and we have not retried the connection        once, disconnect from the server and return to step #4 above.        Note that only one retry is attempted. If the second attempt        fails, execute “GC—Terminate Process”.    -   7. if authentication is not successful, return a message to the        client indicating such. Exit this flow.    -   8. Record the returned cookie and tag along with the username        and IP address of the session for future use in accounting        procedures.

GC—Auth Fini

-   -   1. If there is no authentication cookie recorded for this client        connection, exit this flow.    -   2. If the cookie is “chainer”, zero out the counters for this        session and exit this flow.    -   3. If there is an authentication tag associated with the        connection and there were some amount of bytes delivered to the        client, perform the following logic:        -   A. If a connection to the centralized authentication server            is active, generate an accounting packet necessary to            associate the used bandwidth to the account information            recorded in the “GC—Client Authenticate” flow. If            successful, log this information in a “real-time” logfile            indicating that the account was handled in real-time.        -   B. Otherwise, log the information in a “post-processed”            logfile for processing at a later point in time by external            processes.    -   4. If the connection to the authentication server is active,        send a disconnect message with the cookie, IP, user, and tag of        the connection as recorded by the “GC—Client Authenticate” flow.

GC—Terminate Process

-   -   1. Run all registered flows for termination cleanup.    -   2. Terminate the process.

GC—Client Command DATE

-   -   1. Generate and send the following response to the client:        -   “111 YYYYMMD HH:MM:SS”.

GC—Client Command HELP

-   -   1. Respond with “100 help” followed by GC software version        information.    -   2. Continued response with static help information for the        commands available to the client under the current        authentication state.    -   3. Terminate the response with “.\r\n”.

GC—Client Command MODE

-   -   1. If the mode is “READER”, respond with “200” followed by the        name of the news server site as configured by the operator.    -   2. If the mode is “GC—COMPRESS”, and the connection is        authorized to use this mode, flag the session as able to receive        compressed copies of articles and respond with “200 Compressed        Transfers Enabled”.    -   3. For all other mode requests, respond with “500 syntax error        or unknown command”.

GC—Client Command NEWNEWS

-   -   1. Respond with “502 newnews permission denied (command        disabled)”.

GC—Client Command XPAT

-   -   1. Respond with “501 xpat not supported”.

GC—Client Command QUIT

-   -   1. Respond with “205 goodbye”.    -   2. Close client socket.    -   3. Execute “GC—Terminate Process”.

GC—Client Command AUTHINFO

-   -   1. If less than 2 parameters are passed to AUTHINFO, respond        with “501 bad authentication”.    -   2. If type is USER, record the desired username.    -   3. If type is PASS, record the password.    -   4. If type is not USER and type is not PASS, respond with “501        bad authentication” and exit this flow.    -   5. If both a desired username and a password have not been        received, respond with “381 more authentication required” and        exit this flow.    -   6. Execute “GC—Client Authenticate” with the client's IP        address, desired username, and specified password.

GC—Client Command GROUP

-   -   1. If the group fails to pass basic sanity checks, respond with        “411 no such group” and exit this flow.    -   2. If the client is not allowed (by configuration and        authorization) to see the specified group, return “411 no such        group” and exit this flow.    -   3. Execute “GC—Select Backend Server” with the group in        question.    -   4. Get cached information for the group (including lo and hi        article numbers in the group).    -   5. If the command indicates that we should only respond with        cached information OR the timestamp on the cached information is        less than a configurable amount of time old, execute this logic:        -   A. If there are retention restrictions on this session,            adjust the lo-bound returned to reflect this.        -   B. Respond with “211<article-count><lo-bound><hi-bound>”            where article-count is an estimate count of articles in the            group, and the lo-bound (as modified by 5A above) and            hi-bound are from the cached information on the group.        -   C. Exit this flow.    -   6. Connect to the backend-server and send the GROUP command to        it.    -   7. Process the response and if the response code is not 211,        pass the response back through to the client and exit this flow.    -   8. Parse the response and cache the new lo and hi bounds for the        group in the group's metadata file.    -   9. Pass the server's response back to the client.

GC—Client Command LAST

-   -   1. If the current client article number is less than either the        lo bound on the group OR less than the allowed retention lo        bound, set the current client article number to the allowed        minimum.    -   2. Connect to the backend-server and send a STAT command with        the current client article number. Receive the response and        ignore it.    -   3. Send a LAST command to the server. If the response code is        223:        -   A. Set the current client article number to the article            number indicated.        -   B. Pass the response back to the client and exit this flow.    -   4. Respond to the client with “422 no previous article”.

GC—Client Command NEXT

-   -   1. If the current client article number is less than either the        lo bound on the group OR less than the allowed retention lo        bound, set the current client article number to the allowed        minimum.    -   2. If the current client article number is greater than the hi        bound on the group, set the current client article number to be        the group's hi bound.    -   3. Connect to the backend-server and send a STAT command with        the current client article number. Receive the response and        ignore it.    -   4. Send a NEXT command to the server. If the response code is        223:        -   A. Set the current client article number to the article            number indicated.        -   B. Pass the response back to the client and exit this flow.    -   5. Respond to the client with “422 no next article”.

GC—Client Command STAT

-   -   1. If the argument is non-numerical, select an appropriate        backend server for the currently selected group. Send the STAT        command straight through to the backend server and pass the        response to the client. Exit this flow.    -   2. If the argument is numerical or no argument was passed and        there is no currently selected group, respond with “412 no group        selected” and exit this flow.    -   3. If the argument is numerical, retrieve the variable retention        lo-bound for the current group and if the argument is less than        the retrieved value, respond with “423 no such article in group”        and exit this flow.    -   4. If no argument was given to the command, send “stat #” where        # is the value of the current client article number.    -   5. Else, send “stat #” where # is the value passed as the        argument.    -   6. Return the response from the backend to the client.

GC—Client Command NEWGROUPS

-   -   1. If multi-server mode is not enabled, send the newgroups        command to the backend server and pass the response on unaltered        to the client. Exit this flow.    -   2. Parse the arguments to NEWGROUPS and retrieve the offset into        the cached ACTIVE file for groups created on or after the        specified timestamp.    -   3. Open the ACTIVE file, seek to the specified offset.    -   4. Respond with “231 newgroups follow”    -   5. Read from ACTIVE file until EOF and send all data to the        client.

GC—Client Command POST

-   -   1. Connect to an appropriate backend server for posting (if        postd is enabled, the backend server will be a posting-only        server).    -   2. Send POST to the backend server.    -   3. Respond to client with “340 send article”.    -   4. If postd is enabled:        -   A. Send postd specific headers followed by a blank line.        -   B. Send NNTP-Posting-Host and NNTP-Posting-Date as client            headers.    -   5. Read the headers from the client, for each header perform the        action specified by “GC—Post Header Filter”.    -   6. If any errors occurred during header processing, send “441        posting failed” to the client and discard remaining data from        the client until the end of the post is detected. Disconnect        from the backend server and exit this flow.    -   7. If postd is enabled:        -   A. If no message-id header was sent by the client, generate            a unique message-id and send this header to the backend            server.        -   B. If no date header was sent by the client, generate and            send an appropriate date header to the backend server.    -   8. If postd is not enabled:        -   A. Generate an encrypted trace line called “X-GC-Trace”            including information necessary to allow the secured ability            to identify the true poster.    -   9. Terminate the headers by sending a blank line to the backend        server.    -   10. Pass the remainder of the post (the body of the article)        from the client straight through to the backend server until        terminated with “.\r\n”.    -   11. Return resulting response from backend server to the client.    -   12. Disconnect from the backend server.

GC—Client Command LISTGROUP

-   -   1. If no arguments were passed and there is no current client        group selected, respond with “412 no group selected” and exit        this flow.    -   2. If an argument does exist:        -   A. Execute “GC—Client Command Group” with the specified            argument with the exception that NO responses generated from            the execution of this subflow is allowed to reach the            client.        -   B. If the newly set client group is empty or is            non-existent, respond with “411 no such group”.    -   3. Get cached metadata (including the lo- and hi-bounds) for the        group in question. Adjust the lo-bounds as necessary for any        retention-specific configuration.    -   4. Respond to the client with “211 article numbers follow”.    -   5. Respond with the articles from lo-bound to hi-bound with one        number per line incrementing by 1 on each successive line.    -   6. Terminate the response with “.\r\n”.

GC—Client Command LIST

-   -   1. Initialize listexpire to 900.    -   2. If multi-server mode is enabled:        -   A. If the first argument is ACTIVE, execute “GC—Multi-Server            LIST ACTIVE” with the remaining arguments to it. Exit this            flow.        -   B. If the first argument is ACTIVE.TIMES, set listexpire to            0.    -   3. If there was an argument passed in, execute “GC—Retrieve        Cached List” with the specified argument and listexpire value.        Return the cached data to the client and exit this flow.    -   4. Otherwise, pass the command verbatim to the backend server        and return the response unmodified to the client.

GC—Multi-Server LIST ACTIVE

-   -   1. If there is no argument passed in:        -   A. Execute “GC—Retrieve Cached List” with ACTIVE as the            argument and 0 as the listexpire parameter.        -   B. If there were any errors with cached retrieval, execute            “GC—Terminate Process”.        -   C. Write the cached data to the client.        -   D. Exit this flow.    -   2. If there is no ‘*’ anywhere in the argument passed:        -   A. Retrieve the cached group metadata as well as the cached            moderation flag for the group specified in the argument.        -   B. Generate and send to the client the appropriately            formatted response from the data retrieved.    -   3. Otherwise, open the cached ACTIVE file, send the first line        of the file to the client unconditionally.    -   4. For each line read from the ACTIVE file, perform pattern        matching on the group in the line against the argument passed.        If a match is detected, send the line to the client.    -   5. The last line (the terminating “.\r\n”) is always sent to the        client unconditionally.

GC—Retrieve Cached List

-   -   1. If the requested list is not cached, indicate NODATA and exit        this flow.    -   2. If the timestamp on the cached list is older than listexpire        seconds old and listexpire is not 0, indicate NODATA and exit        this flow.    -   3. Read the cached list into memory and indicate SUCCESS.

GC—Post Header Filter

-   -   1. If postd is enabled:        -   A. If the header is ‘date’, record the fact that the date            header has been sent.        -   B. If the header is ‘message-id’, record the fact that the            message-id header has been sent.        -   C. If the header is NNTP-Posting-Date, NNTP-Posting-Host,            Path, or X-GC-Moderated-Groups; drop the header entry and do            not pass it to the backend server.        -   D. If the header is ‘newgroups’, pull out moderated groups            and create a new header to contain them called            ‘X-GC-Moderated-Groups’.    -   2. If the header is NNTP-Posting-Host or Path, drop the header        entry and do not pass it to the backend server.    -   3. If the header is ‘newgroups’, ensure that the client has        permission to post to those groups; if not, drop the header        entry and do not pass it on to the backend server.

GC—Chop Xover Method

-   -   1. Arguments expected are: range, hdr, pat.    -   2. Get cached metadata on group (including lo- and hi-bounds).    -   3. If passed argument range is ‘seed’, then set the xover lo to        the highest numbered xover article cached locally for the group;        and xover hi to the group hi-bounds.    -   4. Else, if the passed argument range is in the form #, set        xover lo to the number in the argument and xover hi to the group        hi-bounds.    -   5. Else, if the passed argument range is in the form of #-#, set        xover lo to the first number in the argument and xover hi to the        second number in the argument.    -   6. If necessary, adjust xover lo and xover hi to the group lo-        and hi-bounds; also ensure that the xover lo is greater than or        equal to the allowed retention of the group.    -   7. Execute “GC—Select Backend Server”.    -   8. If xover lo is greater than or equal the lowest xover cached        for the group and xover hi is less than or equal to the highest        xover cached for the group:        -   A. Return proper result code for the command executed:            -   i. if (hdr !=NULL) “221% s fields follow (cached)”, hdr            -   ii. if (hdr==NULL) “224 xover information follows                (cached)”        -   B. Execute “GC—Pump Xover Method” with xover lo, xover hi,            hdr, and pat as arguments.        -   C. Send terminating “.\r\n”.        -   D. Exit this flow.    -   9. If xover lo is less than or equal to the highest cached xover        +1 and xover hi is greater than or equal to the lowest cached        xover −1:        -   A. Return proper result code for the command executed:            -   i. if (hdr=NULL) “221% s fields follow (overlap)”, hdr            -   ii. if (hdr==NULL) “224 xover information follows                (overlap)”        -   B. If xover lo is less than lowest xover cached, execute            “GC—Suck Xover Method” with xover lo, lowest xover # cached            −1, hdr, and pat. overlap lo=lowest xover # cached.        -   C. Else overlap lo=xover lo.        -   D. If xover hi>highest xover # cached, then overlap            hi=highest xover # cached.        -   E. Else overlap hi=xover hi.        -   F. Execute “GC—Pump Xover Method” with overlap lo, overlap            hi, hdr, pat.        -   G. If xover hi is greater than the highest xover # cached,            then execute “GC—Suck Xover Method” with highest xover #            cached, xover hi, hdr, and pat.        -   H. Send terminating “.\r\n”.        -   I. Execute this flow.    -   10. If we have never cached anything for this group (lowest and        highest # cached xover are both 0):        -   A. Return proper result code for the command executed:            -   i. if (hdr !=NULL) “221% s fields follow (overlap)”, hdr            -   ii. if (hdr==NULL) “224 xover information follows                (overlap)”        -   B. Execute “GC—Suck Xover Method” with xover lo, xover hi,            hdr, and pat.        -   C. Send terminating “.\r\n”.        -   D. Exit this flow.    -   11. Otherwise, Pass through the client's command verbatim to the        backend server and send the resulting response to the client        unmodified and uncached.

GC—Pump Xover Method

-   -   1. Arguments expected are: lo, hi, hdr, pat.    -   2. b=lo, c=1024.    -   3. Perform the following logic in a loop:        -   A. If b is greater than hi, exit this flow.        -   B. e=b+c.        -   C. If e is greater than hi, e=hi.        -   D. Retrieve the xover from the local cache listing only            those fields specified in hdr, and matching pattern pat for            the range of articles numbered b through e.        -   E. If the data is retrieved without error, write the results            to the client.        -   F. If no data is returned from the local cache and lo equals            hi, execute “GC—Suck Xover Method” with lo, hi, hdr, pat.        -   G. B=e+1.        -   H. Restart the loop at step #3.

GC—Suck Xover Method

-   -   1. Arguments expected are: lo, hi, hdr, pat.    -   2. If remaining drive space on the metadata storage device is        less than 1% of the total drive space, or an attempt at locking        the metadata file for the current client group fails:        -   A. Pass the customer command through to the backend server            and return the results to the client without caching the            data.        -   B. Exit this flow.    -   3. Get the cached group bounds for this group.    -   4. b=lo, c=256.    -   5. Perform the following logic in a loop:        -   A. If b is greater than hi, exit the loop and continue at            step #6.        -   B. e=b+c.        -   C. If e is greater than hi, e=hi.        -   D. Send the request for xover for range b through e to the            backend server.        -   E. Read response code line.        -   F. If code equals 224 and hdr is not NULL:            -   i. Put xover data into local cache database.            -   ii. Retrieve from the local cache database the data                formatted according to hdr and matching pat.            -   iii. Return the retrieved information to the client.        -   G. Else if code==224 and hdr is NULL:            -   i. Pass the information unaltered directly back to the                client.            -   ii. Put the xover data into the local cache database.        -   H. b=e+1        -   I. Restart the loop at step #5.    -   6. If data was written to the xover cache for the group, update        the group's cached xover range in the metadata file for the        group.    -   7. Unlock the metadata file for the group in question.

GC—Client Command Xover

-   -   1. Execute “GC—Chop Xover Method” with the arguments passed,        NULL, NULL.

GC—Client Command XHDR

-   -   1. If the length of the argument=0, respond with “501 bad        command usage” and exit this flow.    -   2. Change the process priority to be lower than normal due to        the excess processing requirements of XHDR.    -   3. If no range was provided, use “0-0” as the range, else range        is the last argument listed in the chain of arguments.    -   4. Execute “GC—Chop Xover Method” with range, hdr, NULL.

GC—Chop Article Method

-   -   1. Parameters are chop and argument.    -   2. If the argument length equals 0:        -   A. If there is no current client group, respond with “412 no            group selected” and exit this flow.        -   B. num=current client article number.        -   C. msgid=message id from the local xover cache database if            available, NULL if not.        -   D. args=num    -   3. Else if there is a currently selected client group and        argument is numerical:        -   A. Validate the number from the argument as being in the            allowed range of articles for the group by group bounds and            allowed group retention. If not, respond with “423 no such            article in group” and exit this flow.        -   B. msgid=message id from the local xover cache database if            available, NULL if not.    -   4. Else:        -   A. num=0        -   B. msgid=arg    -   5. Client current article number=num    -   6. If msgid is not NULL:        -   A. Execute “GC—Get Article From Cache” with msgid        -   B. If the cached retrieval was successful, execute            “GC—Actual Chop Article” with chop and the article data and            exit this flow.    -   7. If msgid is not NULL and num equals 0:        -   A. Cluster status=execute “GC—Cluster Find Article” with            msgid and 0.        -   B. If cluster status indicates that no server cluster server            was found, execute “GC—Find Backend”    -   8. Else if msgid is not NULL and num is not 0, cluster        status=execute “GC—Cluster Find Article” with msgid and −num.    -   9. Else if msgid is NULL, cluster status=execute “GC—Cluster        Find Article” with the current client group and current client        article number.    -   10. If chop is less than 0 (indicating HEAD):        -   A. If cluster status !=NULL, send the HEAD command to the            backend server.            -   i. On success, pass the result back to the client.            -   ii. On failure, connect to the true backend server and                issue the head command and pass back the results to the                client.        -   B. Exit this flow.    -   11. Send ARTICLE command to the backend or cluster server and        cache the article locally by executing “GC—Cache Article”. If        there was a failure in retrieving the ARTICLE and we were        connected to a cluster-server, disconnect and connect to the        true backend server and attempt step 11 again.    -   12. With the article in memory, execute “GC—Actual Chop Article”        with chop and the article data.

GC—Actual Chop Article

-   -   1. Parameters are chop and article data.    -   2. If the article buffer does not contain “\r\n\n\”, respond to        the client with “423 no such article in group” and exit this        flow.    -   3. If chop is less than 0 (indicating HEAD):        -   A. Respond with “221<article-number><msgid>”.        -   B. Send the headers from the in-memory article image to the            client.    -   4. If chop is equal to 0 (indicating ARTICLE):        -   A. Respond with “222<article-number><msgid>”.        -   B. Send the full article from the in-memory article image to            the client.    -   5. If chop is greater than 0 (indicating BODY):        -   A. Respond with “222<article-number><msgid>”.        -   B. Send the body from the in-memory article image to the            client.

GC—Client Command ARTICLE

-   -   1. Execute “GC—Select Backend Server”.    -   2. Execute “GC—Chop Article” with chop=0 and args=arguments to        the command.

GC—Client Command BODY

-   -   1. Execute “GC—Select Backend Server”.    -   2. Execute “GC—Chop Article” with chop=1 and args=arguments to        the command.

GC—Client Command HEAD

-   -   1. Execute “GC—Select Backend Server”.    -   2. Execute “GC—Chop Article” with chop=−1 and args=arguments to        the command.

GC—Cluster Find Article

-   -   See Data Def: Cluster Request Data Structure    -   1. Create request with information provided to function (as        specified in the Data Def).    -   2. Broadcast the request on well known UDP ports in the cluster.    -   3. Wait for a response for a specific amount of time.    -   4. If no response is received in the time allotted, return NULL        (indicating no cluster response received) and exit this flow.    -   5. Receive the response.    -   6. Validating the response is for the current request; if not,        return NULL and exit this flow.    -   7. Change the backend server connection address to be that of        the responding cluster server.    -   8. Return the msgid received from the cluster response.

GC—Find Backend

-   -   1. If multi-server mode is not enabled, exit this flow.    -   2. Perform a STAT command on the currently connected backend        server. If the result is positive, exit this flow.    -   3. Connect to each server in the multi-server list in sequence        and perform a STAT command for the msgid in question.        -   A. If a server returns success, set that server to be the            current backend server and exit this flow.        -   B. If no server response, return indicating NOT FOUND.

GC—Get Cached Article

-   -   1. Expected arguments are group_or_msgid and num.    -   2. If request is by group:num, use the cached xover data to        translate it to a msgid. If the msgid could not be retrieved        from this data, return a failure to find the article and exit        this flow.    -   3. Locate and check for the existence of the metastat file for        the article. If the metadata file does not exist, return failure        to find the article and exit this flow.    -   4. Read the metadata file into memory.    -   5. If the flags in the metadata file indicate the article is        stored in a cyclic (JBOD) spool:        -   A. Validate that the spool is operational, on a failure:            -   i. Delete the article's metadata file.            -   ii. Return failure to find the article and exit this                flow.        -   B. Open the spool file, on failure return failure to find            the article and exit this flow.        -   C. Seek to the offset indicated by the metadata.        -   D. Read the article into memory. On error:            -   i. Execute “GC—Article 10 Error”            -   ii. Return failure to find the article and exit this                flow.        -   E. Insure that artsize [2] is less than sizeof (artid), else            return failure to find the article and exit this flow.    -   F. Verify the msgid from the article data read from the cyclical        spool and verify that it matches the article we are expecting.        If not, return failure to find the article and exit this flow.    -   6. Else, read the article from the metadata file.    -   7. If the flags indicate that the article is compressed,        decompress it in memory. On failure:        -   A. Delete the article's metadata file.        -   B. Return failure to find the article and exit this flow.    -   8. Return success in finding the article.

GC—Cache Article

-   -   1. Expected arguments are article data and length of article        data.    -   2. Create an xover data structure from the headers of the        article; on error, exit this flow.    -   3. If the article had no msgid or the msgid length was less than        3 or the msgid includes a ‘/’, exit this flow.    -   4. Parse the age of the article from the date header or exit        this flow.    -   5. If the age is less than 0 or greater than a configurable        value called ‘max-artdays’, exit this flow.    -   6. If remaining drive space on the metadata storage device is        less than a configurable amount, exit this flow.    -   7. If compression is enabled and the article size is greater        than 32K:        -   A. Add GC_DB_ARTICLE_COMPRESSED to the flags for the            article.        -   B. Compress the article in memory, on error exit this flow.    -   8. Execute “GC—GET Next Cyclic Spool”, if no error:        -   A. Execute “GC—Check Cyclic Spool”, on failure, exit this            flow.        -   B. Add GC_DB_ARTICLE_CYCLIC to the flags on the article.        -   C. Write metadata to both the cyclic spool and the metadata            file for the article.        -   D. Execute “GC—Advance Cyclic Spool”.        -   E. Write the article data to the cyclic spool.    -   9. Else (indicative of “no-cyclical spool” mode):        -   A. Open the article data file on the cache file system.        -   B. Write the article metadata and data to the file.    -   10. Add the xover information gathered from the headers to the        group's cached xover database.    -   11. If the article was not stored to a cyclical storage unit,        append the msgid to the expire file for articles received in the        current time.

GC—Article I/O Error

-   -   1. Log the error.    -   2. If the error was from a cyclical I/O, disable the spool from        further use until human intervention re-enables it.

GC—Get Next Cyclical Spool

-   -   1. If cyclical spools have not been initialized, return and        indicate that there are no cyclical spools available for use.        Exit this flow.    -   2. Calculate the queue identifier corresponding to the age of        the article to be cached.    -   3. If the queue is relatively new and does not have many spools,        attempt to cycle the oldest spool on the system into the current        queue.    -   4. If there are no spools in the queue with available space,        also try to cycle the oldest spool on the system into the queue.    -   5. If there are still no spools in the queue with space        available, return NO SPOOL AVAILABLE and exit this flow.    -   6. Select (in a round-robin fashion) the next spool in the queue        to store the article in. Lock the spool.    -   7. Look in the spool and delete metadata file for any articles        which the new article will overwrite in the spool.    -   8. Return the selected spool.

GC—Check Cyclical Spool

-   -   1. If the spool has been disabled, return FAIL.    -   2. Else, return SUCCESS.

GC—Advance Cyclical Spool

-   -   1. Update the current and next byte pointers in the spool.    -   2. Release the lock on the spool.

GC—Software Start for ‘gc_cluster’

-   -   See Data Def: Cluster Request Data Structure    -   1. Read configuration files and validate for correctness and        authenticity using DSA signature algorithm.    -   2. Install signal handlers for various error conditions and        termination requests.    -   3. Create listening sockets for cluster services.    -   4. Begin processing loop:        -   A. Receive a request.        -   B. If current CPU utilization is greater than a configurable            value, drop the packet and restart the loop at step 4.        -   C. If the packet is from the localhost or from a system not            on our local cluster subnet, drop the packet and restart the            loop at step 4.        -   D. Check for the existence of the article on the local            system by checking for the existence of the metadata file            for the article. If the metadata file is not found, drop the            packet and restart the loop at step 4.        -   E. Generate a response packet using the following values in            the request structure.            -   i. num=port # of GC server            -   ii. seq=seq # from the request            -   iii. buf=msgid of the article        -   F. Unicast the response to the source address/port of the            request.

GC—Software Start for ‘gc_cachesockd’

-   -   1. Read configuration files and validate for correctness and        authenticity using DSA signature algorithm.    -   2. Install signal handlers for various error conditions and        termination requests.    -   3. Create listening sockets for cluster services.    -   4. Begin processing loop (on any error, close the socket and        restart the loop):        -   A. Receive a request.        -   B. If the request is a NEED request:            -   1. Find the queue holding sockets for the requested                IP:port combination. Expire old connections from the                queue.            -   ii. If no queue found, or the queue is empty, send a                NONE response.            -   iii. Else, send a TAKE response with the fd of the                connection. Tag the client connection as PENDING GOT.        -   C. If the request is a TAKE request:            -   i. Find or create a queue for the IP:port combination.            -   ii. Add the fd to the queue if space is available.            -   iii. Send a GOT response to the client.        -   D. If the request is a GOT request:            -   i. If the connection is not tagged as PENDING GOT,                disconnect the client connection.            -   ii. Else close the local copy of the fd and remove the                PENDING GOT. Tag from the connection.        -   E. Restart the loop at Step 4.

As will be recognized by those skilled in the art, the innovativeconcepts described in the present application can be modified and variedover a wide range of applications. Accordingly, the scope of patentedsubject matter should not be limited to any of the specific exemplaryteachings discussed, but is instead defined by the following claims.

1. A UseNet server system comprising: a backend server for storingarticles; a first communication link connected to said backend server; acluster of servers, connected to said first communication link, whereineach server in said cluster of servers is adapted to be in communicationwith the other servers in said cluster of servers, and wherein at leastone of said servers in said cluster of servers is adapted to storeretrieved articles from said backend server when said articles arerequested by a customer; a second communication link adapted to providearticle requests from at least a first customer to said cluster ofservers and adapted to provide at least one of said retrieved articlesto said at least one customer; and said UseNet server system furtheradapted to retrieve stored articles from said at least one server insaid cluster of servers when a first requested article has beenpreviously requested by a second customer and is stored in said at leastone server in said cluster of servers.
 2. The UseNet server system ofclaim 1, wherein a second server of said cluster of servers is adaptedto retrieve said first requested article from said at least one of saidservers in said cluster of servers when said customer requested articlehas already been requested from said backend servers due to a previouscustomer request for said first requested article.
 3. The UseNet serversystem of claim 1, wherein said retrieved articles stored in said atleast one server in said cluster of servers are each stored for a periodof time until more storage space is required.
 4. The UseNet serversystem of claim 1, where said retrieved articles stored in said at leastone server in said cluster of servers are stored in a memory device thatis divided into smaller sized data storage units wherein each datastorage unit is dynamically assigned a time interval such that onlyarticles originally posted within said dynamically assigned timeinterval are stored in each said storage unit.
 5. The UseNet serversystem of claim 1, wherein said customer requests for articles can befulfilled by retrieving said requested articles from said at least oneserver in said cluster of servers for about 20 to 90 percent of saidcustomer requests for articles.
 6. The UseNet server system of claim 1,wherein said first communications link is a TCP/IP communicationsession.
 7. The UseNet server system of claim 1, wherein saidcommunications link uses a Network News Transfer Protocol (NNTP).
 8. TheUseNet server system of claim 1, wherein each said server in saidcluster of servers is adapted to be in communication with the otherservers in said cluster of servers via a network connection.
 9. TheUseNet server system of claim 8, wherein said network connectioncomprises at least one of an wired connection, a wireless connection, anoptical connection, and a satellite connection or link.
 10. The UseNetserver system of claim 1, wherein the second communications link is aTCP/IP session.
 11. The UseNet server system of claim 1, wherein thesecond communications link uses a Network News Transfer Protocol (NNTP).12. The UseNet server system of claim 1, wherein said each server insaid cluster of servers is a commodity server.
 13. The UseNet serversystem of claim 1, wherein said backend server and said cluster ofservers are geographically distant from each other.
 14. An article ordata storage and retrieval system comprising: a plurality of serversforming a server cluster, each said server of said plurality of servershaving storage space for storing articles and data; a communicationnetwork allowing each one of said plurality of servers to communicatewith each other; a backend server comprising storage space for storingarticles, said backend server being in communication with said servercluster via a first communication link; a first server of said pluralityof servers adapted to accept a request for a first article from acustomer; said first server, via said communication network, queriessaid plurality of servers for said first article; if said first articleis found in one of said plurality of servers storage space, said firstarticle is provided to said first server for delivery to said customer;and if said first article is not found in one of said plurality ofserver, said first server requests said first article from said backendserver.
 15. The system of claim 14, wherein said backend server providessaid first article to said first server for delivery to said customerand wherein said first server stores said first article in said firstserver's storage space.
 16. The system of claim 14, wherein said storagespace of each one of said plurality of servers combined provides lessarticle retention then said storage space of said backend server. 17.The system of claim 14, wherein when said first server queries saidplurality of servers for said first article, said first article is foundin one of said plurality of servers at least 20 percent of the time. 18.The system of claim 14, wherein said communication network comprises atleast one of a wired connection, a wireless connection, an opticalconnection, and a satellite connection or link.
 19. The system of claim14, wherein said first communication link is a TCP/IP session.
 20. Thesystem of claim 14, wherein said backend server is geographicallyseparated from said plurality of servers.
 21. The system of claim 14,wherein each said memory space of each said server of said plurality ofservers is logically divided in smaller sized storage units such thateach said storage space is assigned a time interval for storing articlesoriginally posted on a UseNet within said time interval.
 22. The systemof claim 21, wherein said backend server provides said first article tosaid first server for delivery to said customer and said first serverstores said first article in a first storage space that is assigned atime interval that includes a date of said first article.
 23. The systemof claim 21, wherein said backend server provides said first article tosaid first server for delivery to said customer and wherein said firstserver attempts to store said first article in a first storage spacesuch that, if there are only time interval storage spaces having timeintervals newer than a date of said first article, then said firstarticle is not stored in said first server.
 24. The system of claim 21,wherein said backend server provides said first article to said firstserver for delivery to said customer and wherein said first serverattempts to store said first article in a first storage space such that,if there are only time interval storage spaces having time intervalsolder than a date of said first article, then storage space having theoldest time interval is reassigned a time interval that includes saiddate of said first article and said first article is stored therein. 25.A method for providing news services comprising: providing a localnetwork cluster of news servers; caching data and metadata related tonews services with said news servers in said local network cluster;receiving a request for news services form a client associated with saidlocal network cluster; determining whether said requested news servicesare available from said news servers in said local network cluster; ifso, retrieving said requested news services from said news servers insaid local network cluster and providing said requested news services tosaid client directly from said local network cluster; and if not,creating a session to one of at least one backend server(s) to retrievesaid requested news services.
 26. The method of claim 25, wherein saidrequested news services are retrieved from said one backend server in acompressed format.
 27. The method of claim 25, further comprising:storing said retrieved requested news service; providing said retrievedrequested news service to said client; and storing said retrievedrequested news service in at least one of said news servers.